The goal of this two year competitive continuation research proposal is to develop and validate practical methods for demonstrating and improving the predictive validity of licensing and specialty certification examinations. This goal will be accomplished by undertaking a research oriented predictive validity study that continues NCHSR sponsored research on a new Criterion-Referenced Medical Specialty Certification Examination in Emergency Medicine. Specific aims include: 1) developing and demonstrating the reliability and validity of one or more criterion measures of the quality of clinical performance, 2) undertaking a limited pilot study to demonstrate the feasibility and success of a larger definitive predictive validity study, 3) provide definitive empirical evidence of the predictive validity and utility of: a) multiple-choice test scores, b) examiner ratings of clinical competence demonstrated in Simulated Patient Encounters, and c) certification decisions based on test scores and several eligibility requirements, and 4) to compare the results of sophisticated analysis methods with simple statistical and correlational techniques used in previous inconclusive studies. The primary measure of clinical performance proposed to be developed, evaluated, and used to assess the validity of other measures of clinical performance to be collected, involves ratings by expert examiners of stimulated recall of health care provided using charts of selected patients seen by the participant in the past 24-48 hours. The plan is feasible because of the continuing support and active participation of the American Board of Emergency Medicine in this joint research effort. The research will provide empirical evidence, methods and recommendations of value for all medical licensing agencies and specialty certification boards.